• This is a 3-parter. The first commit moves the OpenSSL related defines from from checksum.c to mdigest.h, because I found myself copy/pasting them more than once otherwise.

• The second commit enables parallel computation of MD5 hashes in the block matching get_checksum2() phase. As each blocks' hash is independent, we can process up to 4 blocks simultaneously with SSE2, and 8 blocks with AVX2, leading to a real-world 2x to 6x performance gain / CPU usage reduction (even over OpenSSL-optimized MD5).

However, to make this happen without significant changes to the rest of rsync's codebase, a block prefetcher had to be created. (This whole commit requires --enable-simd as my previous contributions). Full compatibility is maintained with non-SIMD counterparts.

The same mechanism could be used for multithreading checksums as well, but that is beyond the scope of this patch.

• The third commit provides the MD5P8 whole-file checksum. This is an MD5-based checksum but cuts the input stream into 8 independent streams (64-byte interleave), of which the final states are brought together to create the final MD5 checksum. This has the same strengths (and weaknesses) of the normal MD5 checksum, but allows parallel processing, with again a 2x to 6x performance gain.

Both optimized parallel processing is available (--enable-simd) as reference C. MD5P8 is slightly slower on <10kB files due to the additional overhead, but similar to MD5 on larger files without SIMD, and much faster on larger files with SIMD.

Note that get_checksum2() keeps using normal MD5 even if whole-file checksum is MD5P8, because that is parallelized with SIMD anyway if available, and using MD5P8 would just add overhead and quite probably be slower.

Further note that the CSUM_MD5 and CSUM_MD5P8 defines now appear in both checksum.c and simd-checksum-x86_64.cpp, they need to be kept in sync, perhaps moved to a header?

Motivation: though xxhash is now available for rsync, it is not included into the code itself but an external dependency, and by my last evaluation, many distros do not yet come with xxhash included, and thus the distro-included rsync package will be built without xxhash support. That being said, I can imagine that this PR may not be merged due to it not being part of the direction rsync is moving in, I myself need to be using it due to having an uncommon build target, and the code might as well be available to everyone.

The parallel computation of MD5 hashes in get_checksum2() will benefit connections to both recent builds without xxhash as well as older builds of rsync if the block-matching phase applies. If it doesn't lead to a reduction in transfer time due to connection or disk speed limitations, then it will at least massively reduce CPU usage on the supporting client.

The use-case for MD5P8 is more limited, as its usefulness requires both ends to be running an supporting rsync build, but one end not supporting xxhash. If both ends do support xxhash, that should always be the preferred checksum (while MD5P8 can reach gigabytes per second, xxhash is still twice as fast). I only created it as it was a small effort now that parallel MD5 computation was available anyway, and it doesn't have any external dependencies.

I've done some benchmarks for transferring 1GB files between a fast and a slow CPU on 1GbE LAN, compared to normal MD5 usage (all tests already including my previous block size patches and get_checksum1() optimizations):

get_checksum2() MD5 parallelization with MD5 whole-file checksum, both files existing on both ends:

• 33% transfer time reduction
• 52% CPU usage reduction

get_checksum2() MD5 parallelization and MD5P8 whole-file checksum, both files existing on both ends:

• 54% transfer time reduction
• 84% CPU usage reduction

xxhash for both get_checksum2() and whole-file checksum, both files existing on both ends:

• 54% transfer time reduction
• 90% CPU usage reduction

MD5P8, new file:

• 33% transfer time reduction
• 86% CPU usage reduction

xxhash, new file:

• 33% transfer time reduction
• 92% CPU usage reduction

MD5P8, local checksum:

• 83% CPU usage reduction

xxhash, local checksum:

• 94% CPU usage reduction

Obviously these are highly specific to my setup and YMMV. However, my daily syncing of TB's of data is now twice as fast, with average CPU usage down to less than a quarter. xxhash doesn't run ahead much in this case because CPU power while checksumming is no longer the bottleneck after these patches. With even faster network and disks (10GbE + NVMe) xxhash might be twice as fast.